United States Patent n»i 

Gregg et al. 


IIISIIIIIIII 

US005425020A 

[ii] Patent Number: 
[45] Date of Patent: 


5,425,020 
Jun. 13, 1995 


[54] SKEW MEASUREMENT FOR RECEIVING 
FRAME-GROUPS 

[75] Inventors: Thomas A. Gregg, Highland; Joseph 
M. Hoke, Millerton; Patrick J. 
Sngrue, Hyde Park, all of N.Y. 


[73] Assignee: 


[21] 
[22] 

[51] 
[52] 

[58] 
[56] 


AppL No.: 
Filed: 


International Business Machines 
Corporation, Armonk; N.Y. 

147,701 

Nov. 4, 1993 


Int. Q.* H04J 3/14 

U.S. a 370/17; 370/91; 

370/100.1; 370/105 

Field of Search 370/17, 91, 92, 105, 

370/105.3, 100.1 

References Cited 
U.S. PATENT DOCUMENTS 


4,809,259 2/1989 
5,003,558 3/1991 
5,025,458 6/1991 
5,267,240 11/1993 


Jonsson 370/100.1 

Gregg et al 375/108 

Casper ctal 375/114 

Bartow et al „ 371/1 


Primary Examiner— -Douglas W. Olms 
Assistant Examiner— Russell W. Blum 
Attorney, Agent, or Firm — Lynn L. Augspurger 


START 


,1202 


INITIALIZE TABLE 


r 1204 


SEND MEASURE 
FRAME GROUP 


WAIT 
NO 


1 


1206 


FRAME RECEIVED ? 


YES 


1208 


[57] 


ABSTRACT 


A system and method for asynchronously transmitting 
data blocks, in parallel, across multiple fibers in a serial 
manner. Frame groups are provided as a mechanism to 
transmit associated data serially on each fiber and tie the 
data being transmitted together. The frame groups do 
not have sequence numbers, therefore, the receiver 
determines which frames are part of a frame group by 
the arrival times of the individual frames. The transceiv- 
ers for each member of the parallel bus asynchronously 
achieve synchronism from either end of the fiber. A 
receiver of a transceiver determines a relative skew on 
a conductor of a link by performing skew measurements 
on a series of calibration frames sent by a transmitter on 
another end of said link. A skew monitor performs skew 
measurements and includes a plurality of skew measure- 
ment registers for measuring the relative skew between 
frames of a frame group, each register having an input 
from a skew counter, and outputs coupled to a micro- 
processor controlling the monitor. Each register gates 
in the value in the skew counter at the time a link con- 
trol word is received. The receiver can determine that 
one or more frames of the calibration frames have been 
lost and have not exceeded the maximum measureable 
skew limits. 

10 Claims, 11 Drawing Sheets 
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SKEW MEASUREMENT FOR RECEIVING 
FRAME-GROUPS 

FIELD OF THE INVENTION 

This invention relates to computer systems, and par- 
ticularly to a transceiver and method for improved 
skew measurement for receiving frame-groups used in 
highly parallel coupled systems. 

CROSS REFERENCE TO RELATED 
APPLICATIONS 

This application contains subject matter which is 
related to the following U.S. patent applications, and 
while of more general applicability is an improvement 15 
which can be used in connection with these related 
inventions. Many of these referenced applications have 
been published in corresponding European and or Japa- 
nese applications as of the filing date of this application, 
but the numbers thereof have not been identified as of 20 
the filing of this application. They are: 

"Configurable, Recoverable Parallel Bus" by N. G. 
Bartow et al, Ser. No. 07/839,657, filed Feb. 20, 
1992; 

"High Performance Intersystem Communications 25 
For Data Processing Systems" by N. G. Bartow et 

^ al, Ser. No. 07/839,652, filed Feb. 20, 1992; 

"Frame-Group Transmission And Reception For 
Parallel/Serial Buses", by N. G. Barrow, et al, Ser. 
No. 07/839,986, filed Feb. 20, 1992 and now U.S. 30 
Pat No. 5,267,240 granted Nov. 30, 1993; 

"Method and Apparatus for Distributed Locking of 
Shared Data, Employing a Central Coupling Facil- 
ity" by D. A. Elko et al, Ser. No. 07/860,808, filed 
Mar. 30, 1992 and now U.S. Pat. No. 5,339,427 35 
granted Aug. 16, 1994; 

"Communicating Messages Between Processors And 
A Coupling Facility" by D. A. Elko et al, Ser. No. 
07/860,380, filed Mar. 30, 1992; 

"Sysplex Shared Data Coherency Method and 40 
Means" by D. A. Elko et al, Ser. No. 07/860,805, 
filed Mar. 30, 1992; 

"Command Quiesce Function" by D. A. Elko et al, 
Ser. No. 07/860,330, filed Mar. 30, 1992 and now 
U.S. Pat. No. 5,339,405 granted Aug. 16, 1994; 45 

"Storage Management For a Shared Electronic Stor- 
age Cache" by D. A. Elko et al, Ser. No. 
07/860,807, filed Mar. 30, 1992; 

"Management Of Data Movement From A SES 
Cache To DASD" by D. A. Elko et al, Ser: No. 50 
07/860,806, filed Mar. 30, 1992; 

"Command Retry System" by D. A. Elko et al, Ser. 
No. 07/860,378, filed Mar. 30 1992; 

"Integrity Of Data Objects Used To Maintain State 
Information For Shared Data At A Local Com- 55 
plex" by D. A. Elko et al, Ser. No. 07/860,800, 
filed Mar. 30, 1992 and now U.S. Pat. No. 5,331,673 
granted Jul. 19, 1994; 

"Management Of Data Objects Used To Maintain 
State Information For Shared Data At A Local 60 
Complex" by J. A. Frey et al, Ser. No. 07/860,797, 
filed Mar. 30, 1992; 

"Recovery Of Data Objects Used To Maintain State 
Information For Shared Data At A Local Com- 
plex" by J. A. Frey et al, Ser. No. 07/860,647, filed 65 
Mar. 30 1992; 

"Message Path Mechanism For Managing Connec- 
tions Between Processors And A Coupling Facil- 


ity" by D. A. Elko et al, Ser. No. 07/860,846, filed 
Mar. 30 1992; 
"Method And Apparatus For Notification Of State 
Transitions For Shared Lists of Data Entries" by J. 
A. Frey, et al., Ser. No. 07/860,809, filed Mar. 30, 
1992; 

"Method And Apparatus For Performing Condi- 
tional Operations on Externally Shared Data" by J. 
A. Frey, et al., Ser. No. 07/860,655, filed Mar. 30, 
1992; 

"Apparatus And Method For List Management in A 

Coupled DP System" by J. A. Frey, et al., Ser. No. 

07/860,633, filed Mar. 30, 1992; 
"Interdicting I/O And Messaging Operations In A 

Multi-System Complex" by D. A. Elko, et al., Ser. 

No. 07/860,489, filed Mar. 30, 1992; 
"Method and Apparatus for Coupling Data Process- 
ing Systems" by D. A. Elko, et al., Ser. No. 

07/860,803, filed Mar. 30, 1992 and now U.S. Pat. 

No. 5,317,739 granted May 31, 1994. 
The following related applications are unpublished: 
"Quiesce and Unquiesce Function for Intersystem 

Channels", by Neil G. Barrow, et al., Ser. No. 

08/071,154, filed Jun. 1, 1993; 
"Configurable, Recoverable Parallel Bus", by Neil 

G. Barrow et al, Ser. No. 08/071,146, filed Jun. 1, 

1993; 

"Frame Group Transmission and Reception for 
Parallel/Serial Buses", by Neil G. Barrow, et al, 
Ser. No. 08/071,115, filed Jun. 1, 1993; 

"Concurrent Maintenance of Degraded Parallel/- 
Serial Buses", by Kenneth J. Fredericks et al, Ser. 
No. 08/070,587, filed Jun. 1, 1993; 

"Null Words for Pacing Serial Links to Driver and 
Receiver Speeds", by Daniel F. Casper et al, Ser. 
No. 08/071,350, filed Jun. 1, 1993, 

"Error Detection and Recovery in Parallel/Serial 
Busses", by T. A. Gregg, et al, Ser. No. 
08/070,589, filed Jun. 1, 1993. 

"Dumping Service Facility for Multisystem Environ- 
ments", by D. A. Elko, et al, Ser. No. 08/073,909 
filed Jun. 8, 1993. 

The following applications are filed concurrendy 
herewith: 

"Mechanism for Receiving Messages at a Coupling 
Facilty", by D. A. Elko, et al, Ser. No. 08/147,697, 
filed Nov. 4, 1993; 

"Suspending, Resuming, and Interleaving Frame- 
Groups", by T. A. Gregg, et al, Ser. No. 
08/148,113, filed Nov. 4, 1993; and 

"Frame-Group Reception and Processing", by T. A. 
Gregg, et al, Ser. No. 08/147,702, filed Nov. 4, 
1993. 

"A Message-Dispatching Mechanism for Messages 
Received at a Coupling Facility", by D. A. Elko, et 
al, Ser. No. 08/147,703, filed Nov. 4, 1993. These 
co-pending applications and the present application 
are owned by one and the same assignee, Interna- 
tional Business Machines Corporation of Armonk, 
N.Y. 

The descriptions set forth in these co-pending appli- 
cations are hereby incorporated into the present appli- 
cation by this reference. 

BACKGROUND OF THE INVENTION 

Fiber optics enable the transmission of long strings of 
data in a serial fashion from a driver to a receiver at long 
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distances (kilometers) at very high data rates (billions of able physical design should limit this skew to several bit 

bits per second). This is in contrast to traditional electri- times. 

cal wires which allow data to be transmitted for only To avoid data errors of the type which can be caused 

short distances at these data rates (10's of meters). by skew, some means for synchronization of the parallel 

High performance computers require system buses 5 bits is conventionally employed. Traditional parallel 

between elements of a computing system to have even .bus designs use a common clock to achieve synchroni- 

higher band widths (100*s of millions of bytes per sec- zation. This solution, while workable, is inefficient in 

ond). The use of a single fiber optic cable running at its that the common clock typically causes the data collec- 

maximum bandwidth can be insufficient to serve as a tion to be performed in lock step, at a speed limited at 
system bus between elements of a computing system. 10 the upper end by the slowest bit path transmission time. 

In the prior ad, a classical solution to the problem of Thus what is needed is an improved method for trans- 
inadequate bandwidth on a single carrier is to transmit mining data words over a plurality of carriers such as 
the bits of each data word, in parallel, one over each of multiple fiber optic filaments, 
a plurality of carriers (a parallel bus). Another solution, 

known as striping, entails sending a plurality of serial 15 SUMMARY OF THE INVENTION 

data words, in parallel, one over each of a plurality of The present invention comprises a system and 

carriers. These solutions, however, share a common method used in asynchronously transmitting data 

problem in that skew can develop between the parallel blocks, in parallel, across multiple carriers in a serial 

bits or words as they are sent over the bus. manner. Frame groups are provided as a mechanism to 

Skew is a propagation time variation between the 20 transmit associated data serially on each fiber and tie the 
fastest and the slowest carrier. If not handled properly, data being transmitted together. The receiver is pro- 
such propagation time variations can cause data errors vided with a previously measured maximum skew value 
at the receiver. Skew can be caused by differences in the for a given link from either end of the communications 
physical characteristics of the driver, receiver and link. Using this information, the receiver determines 
transmission medium for each carrier of the bus. 25 which frames are part of a frame group by the arrival 

There are two general categories of skew. One is times of the individual frames. This aspect of the inven- 

random skew (RSKEW) and appears as jitter. Sources tion is achieved by a new method and a new transceiver 

of this skew are due to clock synchronization and null which enables the improved method to be performed, 

word insertion. The other type of skew is systemic Thus our present invention has two aspects, a system 

(SSKEW) in that one transceiver is always slower or 30 aspect for the transceiver and a method aspect which 

faster than another by a fixed amount of time. Differ- employs the new hardware and by which the receivers 

ences in carrier propagation times are a typical source on each side of the bus determines the relative skew for 

of the systemic type of skew. each conductor by performing skew measurements on a 

The transmitter, carrier, and receiver each introduce series of calibration frames generated by the transmit- 
skew. Below is a description of how each of these ele- 35 ters on the other side of the bus. Multiple skew measure- 
ments introduces skew. ments registers and a new code algorithm provide a 

Transmission launch skew is the time difference be- greatly faster method of measuring the relative skew, 

tween the start of frame of the earliest transmitter and Each time a skew-measure frame is received, the skew 

the start of frame of the latest transmitter. This type of information for all of the conductors is gathered at the 

skew is measured at the transmitters. There are three 40 same time. This information is then processed for all of 

sources of transmission launch skew. The first is random the conductors by the code. Advantageously, this 

and is caused by the synchronization of the transmitter method greatly reduces the time required to perform 

clock to the system clock. Where the all transmitter the skew measurement. This method is fast enough that 

clocks are locked to a common reference clock, and it can be used in all situations where skew measurement 

there is a common system clock synchronization point, 45 is required, including the more performance critical link 

the launch skew can be as small as one bit time, or even recovery required. Additionally, the random element of 

less. The second source of transmission launch skew is the skew (RSKEW) can also be measured to determine 

also random and is caused by null insertion. The third if the system has been designed correctly, 

source of transmission launch skew is systemic and is The method can be employed in the related inven- 

due to differences in internal signal propagation times 50 tions, with a change to the transceiver receiver section, 

from transceiver to transceiver. Any reasonable physi- which provides a new transceiver for the links between 

cal design should limit this skew to several bit times. two computing elements. This transceiver is used for 

A significant amount of skew is caused by the charac- asynchronously transmitting data blocks, in parallel, 

teristic of the carrier itself. The carrier skew is simply across multiple fibers in a serial manner. Frame groups 

the difference in propagation times between the slowest 55 are provided as a mechanism to transmit associated data 

and fastest carriers in a link. This skew is almost entirely serially on each fiber and tie the data being transmitted 

systemic. Slight variations can occur over time due to together. The frame groups do not have sequence num- 

environmental changes such as temperature. Thus, it bers, therefore, the receiver determines which frames 

should be understood that even if a plurality of frames are part of a frame group by the arrival times of the 

are launched at the same time, they are likely to arrive 60 individual frames. The transceivers for each member of 

at different times because of the carrier skew. the parallel bus asynchronously achieve synchronism 

Skew can also be caused by the characteristics of the from either end of the fiber. A receiver of the trans- 
receiver. There are two sources of skew in the receiver. ceivcr determines which frames are part of a group and 
The first is random and is caused by the synchronization determines a relative skew on a conductor of a link by 
of the system clock to the receiver clock. The second 65 performing skew measurements on a series of calibra- 
source of receiver skew is systemic and is due to differ- tion frames sent by a transmitter on another end of said 
ences in internal signal propagation times between dif- link. A skew monitor performs skew measurements and 
ferent transceivers. As with the transmitter, any reason- includes a plurality of skew measurement registers for 
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measuring the relative skew between frames of a frame 
group, each register having an input from, a skew 
counter, and outputs coupled to a microprocessor con- 
trolling the monitor. Each register gates in the value in 
the skew counter at the time a link control word is 5 
received. The receiver can determine that one or more 
frames of the calibration frames have been lost and have 
not exceeded the maximum measurable skew limits. 

According to an embodiment of the present inven- 
tion, the transceivers for each member of the parallel 10 
bus asynchronously achieve synchronism at each end of 
the fiber. Thus the need for a common clock is elimi- 
nated. The receivers on each side of the bus determine 
the relative skew for each conductor by performing 
skew measurements on a series of calibration frames 15 
generated by the transmitters on the other side of the 
bus. When the skew on all conductors, viewed from 
both sides of the bus, has been determined, the skew 
values are exchanged across the bus, thus enabling the 
transmitters to set proper frame spacing. 20 

Advantageously, skew measurement according to the 
present invention enables data to be sent across a link 
comprising a multi-carrier bus, limited only by the max- 
imum bandwidth of the link itself rather than by the 
worst case theoretical skew. According to an embodi- 25 
ment of the present invention, data integrity is ensured 
by first measuring the relative skew values for the carri- 
ers in a link, at the time that the link is established or 
reinitialized. After the relative skew values have been 
determined, they are exchanged between both sides of 30 
the link and a connection is established having the start- 
to-start spacing of transmitted frames limited to the 
worst case (largest) skew value measured by the receiv- 
ers. Since frames can be made very large, a significant 
amount of data can be sent across the link, at the maxi- 35 
mum link bandwidth without concern for skew. Fur- 
ther, where large data frames are used, the bandwidth 
limiting effect of limiting the start-to-start frame spacing 
to the worst case skew value for a given link can be kept 
at a minimum. 40 

It will be appreciated that the present inventions of a 
transceiver and method for improved skew measure- 
ment for receiving frame-groups used in highly parallel 
coupled systems represents an improvement to the sys- 
tem disclosed in the continuation-in-part of U.S. patent 45 
application Ser. No. 07/839,986 filed Feb. 20, 1992 enti- 
tled "Frame Group Transmission and Reception for 
Parallel/Serial Buses", the continuation application 
being U.S. Ser. No. 08/071,115, filed Jun. 1, 1993. 

BRIEF DESCRIPTION OF THE DRAWINGS 50 

The present invention will be better understood by 
reference to the drawings in which: 

FIG. 1 is a block diagram of a physical link between 
two computing elements; 55 

FIG. 2 is an illustration of the format of an exemplary 
frame; 

FIG. 3 is an illustration of the format of an exemplary 
frame group; 

FIG. 4 is a logic diagram of an exemplary transmitter; 60 
FIG. 5 is a logic diagram of an exemplary receiver; 
FIG. 5A is a logic diagram of an exemplary receiver 

with additional skew measurement registers, illustrating 

our improved transceiver receiver. 
FIG. 6 is a flow diagram of a skew test procedure; 65 
FIG. 7 is a flow chart summary of the skew test- 

/measurement procedure as view from a receiving end 

of the link; 
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FIG. 8 is a flow chart of frame reception and frame 
group integrity checking in a receiver according to an 
embodiment of the present invention; 

FIG. 9 is a flow diagram of an alternative skew test 
procedure. Like reference numerals appearing in more 
than one FIGURE designate like elements; 

FIG. 10 is a flow chart summary of an alternative 
skew test/measurement procedure as view from a re- 
ceiving end of the link; 

The transceiver receiver in monitoring and process- 
ing skew in the process of skew measurement utilizes a 
new method employed in the transceiver receiver, as 
illustrated by FIGS. 11-13B. In these FIGURES: 

FIG. 11 is a diagram of the skew table used in the 
improved skew measurement procedure; 

FIG. 12 is a flow chart of the improved skew mea- 
surement procedure; 

FIGS. 13A and 13B show detailed flowcharts of the 
skew table update and the skew table final processing 
used in the improved skew measurement procedure. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

Before describing particularly the improvements 
which result in a new transceiver and method for skew 
measurement, the environment of the related applica- 
tions will be described. 

Turning first to FIG. 1, a physical link between two 
computing elements 102, 104 is illustrated. These ele- 
ments could be, for example, two computers or a com- 
puter and a shared memory device. In any event, the 
computing elements 102, 104 are connected by way of 
an intersystem channel link 106 comprising a fiber optic 
bus 108, The fiber optic bus 108 is formed of multiple 
fiber pairs 110. Each fiber pair consists of two optical 
fibers, one for transmitting information and one for 
receiving information. While any number of fiber pairs 
can be used, exemplary values for intersystem channels 
would be a range of 1 to 64 fiber pairs in powers of two. 
The fiber pairs 110 of the fiber optic bus 108 are coupled 
to the computing elements 102, 104 by way of transceiv- 
ers 112, 114 located at opposite ends bus. Each of the 
transceivers 112, 114 includes a transmitter unit and a 
receiver unit, both of which will be described in more 
detail later. 

Data is sent across the link from an originator buffer 
116 in a transmitting computing element. The received 
data is then collected and temporarily stored in a recipi- 
ent buffer 118 in a receiving computing element. Al- 
though one buffer is shown in each of the computing 
elements of FIG. 1, it should be understood that a com- 
puting element can dynamically set up various numbers 
of originator and/or recipient buffers depending on the 
number and type of links to be established. The buffers 
116, 118 can be of the "store-thru" type, meaning that 
they need not be filled before they are unloaded. This 
type of buffering has the advantage of lower transmis- 
sion and reception latency times. 

The information to be sent is divided up among the 
fiber pairs 110 that form the fiber optic bus 108. The 
information is sent from the originator buffer 116 to the 
recipient buffer 118 in the form of frames. Each frame is 
transferred on a single fiber of the bus 108. More infor- 
mation on configuration of the link and the structure of 
the buffers are contained, respectively, in copending 
applications entitled CONFIGURABLE, RECOVER- 
ABLE PARALLEL BUS to Barrow et al. (Ser. No. 
07/839,657) and HIGH PERFORMANCE CHAN- 
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NELS FOR DATA PROCESSING SYSTEMS to number bytes actually received with the total transmit- 

Barrow et aJ. Ser. No. 07/839,652), both filed on Feb. ted byte count which is included as part of every LC. 

20, 1992, referenced above. As an alternative embodiment, the transmitter can 

The format of an exemplary frame is illustrated in ensure that the link control words in each frame group 

FIG. 2. Each frame starts with a link-control word 5 are identical with each other, but are different than 

(LC) 202. Various fields in the link-control word 202 those in the immediately previous and subsequent frame 

identify the frame format and type, designate a buffer groups. 

area, and control the state of the transceiver and link. Information from an originator buffer is sent across 
A link-control-CRC (cyclical redundancy check) the link in one or more frame groups. When several 
word 204 follows the link-control word. The link-con- 10 frame groups are used to transmit the contents of one 
trol CRC word 204 is conventionally generated from buffer, the frame groups may be sent in a sequence 
the values in the link-control word. The link-control corresponding to the ascending order of the words in 
CRC word is checked at the receiver to test the validity the buffer. For example, groups of lower numbered 
of the link-control word in the incoming frame. words are sent before groups of higher numbered 
There are two types of frames, control frames and 15 words. When the operational link comprises more than 
information frames. Control frames do not require an one transceiver, the buffer contents are sent in units of 
information field. They consist only of a link-control words on a transceiver basis. That is to say, the words 
word and a link-control CRC word. An information are distributed across the transceivers (and thus the 
frame has a link-control word 202 a link-control CRC fibers), modulo the number of transceivers in the link 
word 204 and an information field 206. Information 20 (e.g. transceiver 0 has word 0, 4, 8, 12 in a four trans- 
fields contain, for example, from one to 1,024 words. ceiver operational link). 

The information field is used to send information from a According to an embodiment of the present inven- 

buffer area at one end of the link to a buffer area at the tion, the transceivers for each carrier of the bus asyn- 

other end. chronously achieve synchronism at each end of the 

An information field is followed by an information- 25 fiber. Thus the need for a common clock is eliminated, 

field CRC word 208. The information-field CRC word Various means for synchronizing frames transmitted as 

is conventionally generated from the values in the infor- serial data over a signal conductor are known in the art 

mation field. The information field CRC word is and thus will not be discussed in detail here. An example 

checked at the receiver to test the validity of the infor- of such synchronization means is described in U.S. Pat. 

mation field in the incoming frame. 30 No. 4,970,714 to Chen et al. and assigned to the same 

Related information can be placed in several frames, assignee as the present invention, 
one on each transceiver of the operational link. These Each fiber and its associated transmission and recep- 
frames (referred to as a "frame group") are transmitted tion hardware has differing propagation characteristics 
concurrently. There can be as many frames in the group which result in skew. Skew is the propagation time 
as there are transceivers in the operational link. FIG. 3 35 variation between the fastest and the slowest fiber. Re- 
shows an exemplary frame group 302. The use of the ception of a start of a frame on any fiber causes the 
frame group enables the data to be sent at a bandwidth receiver to start checking for reception of the complete 
that is multiples of the single fiber optic bandwidth since frame group and to reassemble the received frames of 
multiple frames (one per fiber) are transmitted simulta- data in the recipient buffer. The receiver determines 
neously across the link. 40 which frames are part of a frame group by the arrival 

Although not provided with sequence numbers, the times of the individual frames and the contents of the 
link control words in each frame group are interlocked link control words. If all of the frames of a frame group 
with those of subsequent and previous frame groups. are not received within an established skew window or 
The interlocking can be accomplished in a number of if the link control words received within the skew win- 
different ways. For example, each transmission may 45 dow are not identical, a skew error is recognized, 
start with a request, followed by data, followed by an The skew rules, according to an embodiment of the 
acknowledge. Each of these types of transmissions hav- present invention, set the skew window in which all 
ing a unique link control word. Further, a sending and- frames of a frame-group must be received in order to 
/or recipient buffer address can be sent within the link receive a valid message. The rules also set the rate at 
control words. Thus data transmissions from (and/or 50 which consecutive frame groups can be transmitted so 
directed to) different buffers will have different link that frames of one frame-group are not confused with 
control words. frames of another frame-group. Further, examination of 

In the preferred embodiment, frames associated with link control words for consistency within a given frame 
the same buffer and the same type of frame group may group protects the system from "lost frame" effects, 
have the same link control words. For example a given 55 For a given link, the skew of frame groups measured 
message may include 16 data frames, 15 of which will at the receiver varies by plus or minus the random skew 
have the same LC and one of which will have a distinct (RSKEW) component. The maximum RSKEW com- 
LC in that it includes a bit indicating the start of a buffer ponent is set by the design of the link hardware while 
area. In this case, a lost frame among the 15 having an the systemic skew (SSKEW) component is largely due 
identical LC can not necessarily be detected by compar- 60 to the optical cable installation. According to an em- 
ing the LCs in each received frame group. Instead, bodiment of the present invention, a measurement of the 
either the comparison of LCs in the last frame group total skew (the optical and random components) is 
will detect the problem because a frame belonging to made at the time the link is initialized to ensure that the 
the next frame group (of a different type and having a it does not exceed set limits and further to ensure that 
different LC) has been received, or the skew timer will 65 frames belonging to a common frame group are prop- 
expire (as described later). In either event, the receiver erly identified as such by the receiver, 
will detect that all data has not been received for the An exemplary transmitter unit according to an em- 
buffer area. This is accomplished by comparing the total bodiment of the present invention is illustrated in FIG. 
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4. A four transceiver link is shown. The individual new LC, the corresponding S-R latch 528-534 is set 
transmitters (one for each fiber optic carrier) are shown The four S-R latches 528-534 control the MPX 564, and 
as XMITO 402, XMIT1 404, XMIT2 406, and XMIT3 are inputs to the first-in first-out buffer (FIFO) 568, a 
408. It should be understood that the transmitter of four input AND gate (A) 538, and a four input OR gate 
FIG. 4 can be operated using fewer than four carriers 5 (O) 536. The four OR gates 570-576 each have two 
(e.g. 3, 2 or 1) by gating out selected transmit portions. inputs. One input is a line 510 (A-D) from a correspond- 
The internals of XMITO 402 are shown. Along with ing one of the receivers, and the other input is the out- 
the data paths (not shown), each transmitter has an put of a corresponding one of the S-R latches 528-534. 
input named START 410 which starts frame genera- The output of these OR gates 570-576 signal to the 
tion. Each transmitter also has an output called 10 COMPARE 562 and determines which of the data 
REA R Y . 412 which indicates that the current frame paths to compare. 

transmission has ended, and that the skew wait time has The skew measurement function is performed by a 
ela P sed - counter (SCNTR) 518 which is fed from a second multi- 

Within each transmitter there is a WAIT register 414 plexor (MPX) 512. The second multiplexor 512 gates 
which is initialized at link initialization time. The value 15 either a zero value 514 or the incremented value (+1) 
in the WAIT register 414 is the number of words which 516 to the skew counter (SCNTR) 518. The SCNTR 
must be transmitted between LC (Link Control) words. 518 is an eight bit register. The output of the SCNTR 
The Wait register 414 is connected to a 3 input multi- 518 feeds an equal-to compare circuit (=) 588 and the 
plexor (MPX) 416. The output of the MPX 416 is con- incrementer (+ 1) 516. The other input to compare 588 
nected to a wait counter register (WCNTR) 418. The 20 is from the output of a register (SKEW) 594. When the 
output of the WCNTR 418 is connected to decrementer value in SCNTR 518 is equal to the value in SKEW 594, 
(- 1) 420, an 8 input nand circuit (=0) 422 detecting a the output of the compare circuit 588 is active. The 
value of zero in the WCNTR 418, and back to a first one value in SKEW 594 is set by a microprocessor (not 
(A) of the MPX 416 inputs (A-C). The output of decre- shown) over line 596. 

menter 420 is connected to the third data input (C) of 25 The FIFO 568 stores an LC selected from one of the 
the MPX 416. receivers along with summary information describing 

The channel starts a frame when all four transmitters which of the receivers detected the LC. The write line 
are ready (Le. all transmitters are idle and the minimum 586 to the FIFO 568 is controlled by a four input or (O) 
LC word to LC word spacing time has elapsed) by circuit 598. The microprocessor reads entries in the 
turning on the START line 410 for 1 cycle. This line 30 FIFO 568 over a data bus 590 by activating the read line 
causes each transmitter to begin frame transmission. 584. 

When the Outbound Frame State Machine (OFSM) 424 As with the transmitter unit, receivers for any of the 
sends the LC (indicated on line 426), the END 428 and carriers can be gated out of the system under control of 
MET 430 Shift Register Latches (SRLs) are reset, and the microprocessor via a control register (not shown), 
the value in the WAIT register 414 is gated through the 35 Operation of an exemplary receiver unit according to 
MPX 416 and the WCNTR 418. As words are sent on an embodiment of the present invention as illustrated in 
the link, the Data Request line (DR) 432 is active, and FIG. 5 starts with the receipt of an LC by at least one 
the WCNTR 418 is decremented via the decrementer of the receivers and finishes with a write operation to 
420 and MPX 416 circuits. The End Of Frame (EOF) the FIFO 568. When the first LC is received by one or 
may become active before, during, or after the 40 more of the receivers RCV0 502, RCV1 504, RCV2 
WCNTR 418 reaches the value of zero. The EOF line 506, and RCV3 508, the corresponding S-R latch 
434 sets the END SRL 428, and the 8 input nand circuit 528-534 is set. The output of OR circuit 536 is activated 
422 sets the MET SRL 430. When both SRLs 428, 430 which starts the increment of the SCNTR 518. When all 
are on, the output of the AND circuit 436 is active, and four LCs are received, all four S-R latches 528-534 are 
all conditions have been satisfied for the start of the next 45 set and the output of AND gate 538 is active (HIGH), 
frame for that transceiver. This output feeds an inverter 580 which, in turn, feeds 

The READY lines 412 from the transmitters set the an AND gate 540. Thus, when AND gate 538 is active 
RDY0 438, RDY1 440, RDY2 442, and RDY3 444 (HIGH) the output of AND gate 540 is deactivated 
SRLs. The outputs of these SRLs are connected to a 4 (LOW) which, in turn, stops the SCNTR 518 through 
input AND circuit 446. When all 4 RDY SRLs are set, 50 AND gate 540 and OR gate 599. If all LCs have been 
the output of the AND circuit 446 indicates that all 4 received and they all compare equal (as detected by 
transmitters are ready to start the next frame. The out- COMPARE 562) the output of AND gate 582 is active, 
put of this AND circuit 446 is also used to reset the This output feeds OR gate 598 which, in turn, causes a 
RDY SRLs 438-444. write into the FIFO 568. Along with the LC from the 

An exemplary receiver unit according to an embodi- 55 MPX 564, summary information describing which re- 
ment of the present invention is illustrated in FIG. 5. A ceivers have an LC is also stored in the FIFO 568. 
four transceiver link is shown. The receivers are shown If an LC is damaged by transmission noise on the link, 
as RCV0 502, RCV1 504, RCV2 506, and RCV3 508. it is not detected by the receiver. When this happens the 
The data paths 560 (A-D) from each receiver contain output of AND circuit 538 does not activate, and the 
the current LC detected by the receiver. The output 60 SCNTR continues to count until its value reaches the 
lines 510 (A-D) are active for 1 cycle each time a new value in SKEW 594. At this point, the output of the 
LC is detected by the corresponding receiver. The compare circuit 588 becomes active and it feeds OR 
contents of the data paths 560 (A-D) are compared by circuit 598 which causes an entry to be made into the 
the gated compare logic (COMPARE) 562, and are also FIFO 568. In this situation, the summary information in 
selected by the gated multiplexor (MPX) 564. 65 the FIFO 568 indicates which of the receivers did not 

The Set-Reset latches (S-R latches) V0 528, VI 530, receive an LC. 
V2 532, and V3 534 are set by the respective lines 510 Another possible result of a damaged LC is the appar- 
(A-D) from the receivers. When a receiver detects a ent overlap of two different frame groups. Suppose that 
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two frame groups reach the receivers with the mini- becomes the master (i.e. controls the skew measurement 
mum LC to LC spacing, and suppose that the first frame procedure) and the node with the lower address be- 
of the first group is damaged and discarded. In this case, comes the slave. If only one node has sent the STR 
the activation of the output of OR gate 536 is delayed frame, it becomes the master and the recipient becomes 
until the next LC of the first frame group is received. 5 the slave notwithstanding the relative node address. 
This delay also starts the SCNTR 518 later. After all of Once a skew-test-request frame is received by any 
the undamaged LCs have been received, the SCNTR one of the receivers of the slave receiver unit (and the 
518 is still running. At this point the first frame of the master slave relationship is established), the slave fe- 
second frame group may arrive. If the LC of this frame ceiver unit places itself into a skew measurement mode 
is different that the other LCs at the input of COM- 10 (thereby causing its internal logic to respond as de- 
PARE 562, the output of COMPARE 562 will activate scribed below), resets its skew counter value to zero, 
(since there is a non-compare of the LCs indicating that and sets the skew register 594 to a predetermined initial 
the LCs are from two different frame groups). An value (e.g. 255). It should be understood, however, that 
inverter (N) 566 detects the non-compare and feeds OR another initial value could alternatively be chosen based 
gate 598 which, in turn, causes a entry into the FIFO 15 on actual experience with a given system. The slave 
568. The summary information indicates the LC and the node then sends back a replicated skew-test-response 
receivers which detected this LC. The action of OR control frame 604. This frame indicates to the master 
gates 570-576 bypasses the delay caused by the S-R that the slave is ready to begin the skew measurement 
latches 528-534 so that the LC which caused the non- procedure. 

compare is not included in the summary information. 20 In response to receipt of a skew-test-response frame, 

OR gate 592 receives inputs from AND gates the master receiver unit also places itself into skew 
544-550. Each of the AND gates 544-550 detects if its measurement mode, resets its own skew counter to 
respective receiver receives an LC before the previous zero, loads its skew register 594 with an initial value 
LC (from the same receiver) has been entered into the (e.g. 255) and sends out a replicated skew-measure- 
FIFO 568. Specifically, the AND gates 544-550 detect 25 request (SMR) control frame 606. 
when a new LC is received (as indicated by lines When the first slave receivers) 502-508 receive the 
510A-D respectively) before the S-R latches 528-534 first skew-measure-request frame, the slave receiver 
have been reset. This situation can occur only if the unit determines whether all of the frames have arrived 
transmitter feeding the receivers does not obey the within the skew value indicated by the skew register 
skew rules established at initialization time. The output 30 594. If not, the receiver unit's microprocessor checks 
of OR gate 592 feeds OR gate 598 which causes an the summary information in the FIFO 568 to determine 
entry to be made into the FIFO 568. The summary which frame(s) were late (i.e. not received within the 
information includes the output of OR gate 592. skew value indicated by the skew register 594). The 

In all of the above situations, each time a FIFO 568 receivers detecting the late frames are gated out and are 
entry is made, the S-R latches 528-534 are reset. In the 35 not used in subsequent skew tests. The identity of these 
case of the LC non-compare, the S-R latch for the re- receivers and the skew register value is stored in the 
ceiver which caused the non-compare is set even microprocessors local storage for later use. Next, a new 
though the reset line is active. This is so because the S-R value (e.g. 254) is loaded into the skew register and then 
latches* set function overrides the reset function. Also, the slave sends a replicated skew-measure-response 
each time a FIFO 568 entry is made, the SCNTR 518 is 40 control frame across the link on all carriers (including 
reset to zero by the action of OR gate 599 and MPX those whose receivers have been gated out). 
512. The skew-measure-response frame has the same ef- 

It is noted that the transmitters 402-408 of FIG. 4 are feet on the master receiver unit as the skew-measure- 
not synchronized by a common clock. Similarly, the request frame has on the slave receiver unit. Specifi- 
receivers 502-508 are also not synchronized with one 45 cally, the master determines which frames were re- 
another by a common clock. Instead, each time the link ceived within the skew value indicated by its skew 
is established the transmitter and receiver pairs achieve register 594, outgates the late receivers and loads a new 
synchronization with each other in an asynchronous skew value in its skew register. When this has been 
manner, just as if they connected by the only carrier in accomplished, the master sends a replicated skew-meas- 
the link. Further, the receivers 502-508 use reception of 50 ure-request frame across the link on all carriers, 
the frames themselves, the skew rules and LC checking In response to this skew-measure-request frame, the 
as described above to ensure that frames within the slave receiver unit determines whether all of the frames 
same frame group are properly identified as such. Ad- (received on receivers that have not been gated out) 
vantageously, this eliminates the need for synchronizing have arrived within the new skew value indicated by 
hardware. 55 the skew register 594. If not, the slave receiver unit's 

A flow diagram of a skew test/measurement proce- microprocessor again checks the summary information 
dure according to an embodiment of the present inven- in the FIFO 568 to determine which frame group(s) 
tion is illustrated in FIG. 6. were late. The receivers detecting the late frames are 

When skew measurements are to be performed, any gated out and are not used in subsequent skew tests, 
transceiver may send a skew-test-request (STR) control 60 Also, the identity of the gated out receivers is also 
frame 602 (or a replicated STR control frame) out on stored in the microprocessors local storage for later use. 
the link. By replicated, it is meant that an identical copy Next, a new value (e.g. 253) is loaded into the skew 
of a given frame (e.g. the STR frame) is transmitted on register and then the slave sends a replicated skew mea- 
each carrier within a 2.5 word time transmit window. sure response control frame across the link on all earn- 
The skew-test-request frame includes a link control 65 ers (including those whose receivers have been gated 
word identifying it as an STR. Nodes on both sides of out). 

the link may send out a skew test request at the same The above described process continues iteratively 
time. In this case, the node with the higher address (e.g. 614) until a relative skew value has been deter- 
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mined for each receiver on both sides of the link (i.e. link control word identifying it as an SMR. Only the 

until all but one receiver has been gated out or the skew node that is the master may send out a skew measure 

register has been decremented to zero). When the slave request. The master may be determined by being the 

has measured the relative skew for all receivers, it sets a node with the higher address (i.e. controls the skew 

bit in the skew measure response frame 616 indicating 5 measurement procedure) and the node with the lower 

that the measurement is complete. When the master address becomes the slave, or may be assigned by design 

determines that skew measurements on both sides of the conventions. 

link are complete, it sends a skew-results-request (SRR) The first time that a skew-measure-request frame is 
control frame 622 across the link. In response, the slave received by any one of the receivers of the slave re- 
sends a skew results response control frame 624 (one on 10 ceiver unit (and the master slave relationship is estab- 
each carrier) back across the link. These frames include lished), the slave receiver unit places itself into a skew 
the measured skew values for the slave receivers (e.g. measurement mode (thereby causing its internal logic to 
carrier 0=1, carrier 1=0, carrier 2=6, carrier 3=2). respond as described below), resets its skew counter 
Each of the skew values is indicative of a number of value to zero, and sets the skew register 594 to a prede- 
word transmit times (e.g. 2=2 word transmit times). It 15 termined initial value (e.g. 255). It should be under- 
should be understood that the skew measurements are stood, however, that another initial value could alterna- 
all made relative to the fastest carriers). Thus the fast- tively be chosen based on actual experience with a 
est carrier(s) (i.e. the first to arrive across the link) will given system. The slave node then sends back a repli- 
always have a relative skew value of zero. The skew cated skew-measure-response control frame 908. This 
value returned in the skew results response may also 20 frame indicates to the master that the slave is ready to 
include a padding value added to each measurement. begin the skew measurement procedure. 
This padding value can be used to compensate for ran- When the master unit sends the first skew-measure- 
dom skew (jitter). request frame, the master receiver unit also places itself 

Once the skew-results-response frame has been re- into skew measurement mode, resets its own skew 

ceived, the master can send an establish-transceiver- 25 counter to zero, loads its skew register 594 with an 

group (ETG) control frame 626. This request is used by initial value (e.g. 255) and sends out the first replicated 

the originating (master) node to elicit responses from skew-measure-request (SMR) control frame 910. 

the set of transceivers at the recipient (slave) node and When the first slave receivers) 502-508 receive the 

also to set the intended link (the set of carriers that will next skew-measure-request frame, the slave receiver 

be used to establish communications) at the slave. The 30 unit determines whether all of the frames have arrived 

request is sent as a replicated frame on all of the carriers within the skew value indicated by the skew register 

that have passed the skew test (e.g. have a relative skew 594. If not, the receiver unit's microprocessor checks 

value of 254 or less). The parameter field of the ETG the summary information in the FIFO 568 to determine 

request contains the largest of the skew values deter- which frame(s) were late (i.e. not received within the 

mined by the master receivers. The master can also add 35 skew value indicated by the skew register 594). The 

a pad value in a similar manner as the slave (in sending receivers detecting the late frames are gated out and are 

the skew results response). This value determines the not used in subsequent skew tests. The identity of these 

link-control-word to link-control-word spacing that is receivers and the skew register value is stored in the 

required for any frame groups that are to be sent by the microprocessors local storage for later use. Next, a new 

slave. 40 value (e.g. 254) is loaded into the skew register and then 

In response to the ETG request, the slave sends an the slave sends a replicated skew-measure-response 
ETG response frame 628 (which is replicated on carri- control frame across the link on all carriers (including 
ers on which the ETG request was received). The mas- those whose receivers have been gated out), 
ter node checks the ETG response to ensure that it has The skew-measure-response frame has the same ef- 
been received on each of the carriers on which the ETG 45 feet on the master receiver unit as the skew-measurc- 
request was sent If the master node receives the ETG request frame has on the slave receiver unit. Specifi- 
responses an all transceivers of the intended link, the cally, the master determines which frames were re- 
link is established using the carriers designated by the ceived within the skew value indicated by its skew 
ETG frames. If not, a link can be retried or established register 594, outgates the late receivers and loads a new 
using a subset of the carriers comprising those which 50 skew value in its skew register. When this has been 
are operational. In that event, the master sends another accomplished, the master sends a replicated skew-meas- 
skew results request to the slave and the slave again ure-request frame across the link on all carriers, 
responds with the previously measured skew. Only the In response to this skew-measure-rcquest frame, the 
maximum skew on the intended link carriers is used to slave receiver unit determines whether all of the frames 
determine the minimum frame spacing. 55 (received on receivers that have not been gated out) 

A flow diagram of an alternative skew test/measure- have arrived within the new skew value indicated by 

ment procedure according to an embodiment of the the skew register 594. If not, the slave receiver unit's 

present invention is illustrated in FIG. 9. Note that ND microprocessor again checks the summary information 

is Node Descriptor. This procedure enables measure- in the FIFO 568 to determine which frame group(s) 

ment of skew from either end of the communications 60 were late. The receivers detecting the late frames are 

link. gated out and are not used in subsequent skew tests. 

When skew measurements are to be performed, a Also, the identity of the gated out receivers is also 
transceiver may send a skew-measure-request (SMR) stored in the microprocessors local storage for later use- 
control frame 906 (or a replicated SMR control frame) Next, a new value (e.g. 253) is loaded into the skew 
out on the link. By replicated, it is meant that an identi- 65 register and then the slave sends a replicated skew mea- 
cal copy of a given frame (e.g. the SMR frame)is trans- sure response control frame across the link on all carri- 
mitted on each carrier within a 2.5 word time transmit ers (including those whose receivers have been gated 
window. The skew-measure-request frame includes a out). 
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The above described process continues iteratively measurement mode, in steps 702 and 1002 it measures 
(e.g, 914) until a relative skew value has been deter- the relative skew for all carriers on the bus. Once the 
mined for each receiver on both sides of the link (i.e. measurements have been accomplished on both sides of 
until all but one receiver has been gated out or the skew the link, in steps 704 and 1004 the measured worst case 
register has been decremented to zero). When the slave 5 skew values are exchanged across the link. In the case 
has measured the relative skew for all receivers, it sets a of a slave node, the measured skew values for each 
bit in the skew measure response frame 916 indicating carrier are sent across the link to the master. Finally, in 
that the measurement is complete. When the master steps 706 and 1006, once the exchange is complete, the 
determines that skew measurements on both sides of the receiver unit ends skew test/measurement mode and 
link are complete, it sends a skew-results-request (SRR) 10 awaits a command from the system or link, 
control frame 922 across the link. In response, the slave A summary of the method used by the receiver unit 
sends a skew results response control frame 924 (one on to receive and verify a frame group is illustrated in FIG. 
each carrier) back across the link. These frames include 8. In step 802 the receiver unit receives the first frame or 
the measured skew values for the slave receivers (e.g. frames in a frame group. In response, in step 804 the 
carrier 0=1, carrier 1=0, carrier 2=6, carrier 3=2) 15 receiver unit starts its skew counter to measure elapsed 
and master receivers (e.g. carrier 0=1, carrier 1=2, word times and continuously compares the elapsed time 
earner 2=0, carrier 3=1). Each of the skew values is against the maximum measured skew value for the es- 
mdicative of a number of word transmit times (e.g. 2=2 tablished link. If at any time before all frames have been 
word transmit times). It should be understood that the received (or an error has been detected), the elapsed 
skew measurements are all made relative to the fastest 20 time matches the worst case skew value (as detected in 
carriers). Thus the fastest carriers) (i.e. the first to step 806), in step 808 the receiver unit is reset to receive 
arrive across the link) will always have a relative skew a new frame group (which includes resetting the skew 
value of zero. The skew value returned in the skew counter) and an error is reported, 
results may also include a padding value added to each In parallel, in step 810, as frames arrive their link 
measurement. This padding value can be used to com- 25 control words are compared to ensure that they are all 
pensate for random skew (jitter). This value determines identical. If any newly arrived link control word does 
the link-control-word to link-control-word spacing that not match those for the other received frames (detected 
is required for any frame groups that are to be sent by m step 812) in step 808 an error is signaled and the 
the master. receiver unit is reset to receive a new frame group. The 

Once the skew-results-request frame has been re- 30 non-comparing frames are considered to be part of the 
ceived, the slave tests the skew results to determine the new frame group and are treated by the receiver as 
set of transceivers that are within the maximum allowed such. 

skew at both ends of the link and comprise an allowed If the newly arrived frames compare, a test is made in 
intended operational link. The set of transceivers that step 814 to determine whether all the frames in the 
meets this test becomes the operational link. The slave 35 frame group have been received. This is a matter of 
can indicate non-acceptance of an operational link by a comparing the active carriers, as indicated by the re- 
bit in the parameter field. ceiver unit's control register, with the received and 

Once the skew-results-response frame has been re- properly comparing frames. If all frames have not yet 
ceived, the master tests the skew results to determine been received, in step 816 the receiver continues to wait 
the set of transceivers that are within the maximum 40 for additional frames to be received and continues the 
allowed skew at both ends of the link and comprise an comparison and time measurement process as described 
allowed intended operational link. The set of transceiv- above. If all frames have been received before the worst 
ers that meets this test becomes the operational link. case measured skew time has elapsed, the receiving end 
The slave can indicate non-acceptance of the opera- of the link signals complete and the receiver is reset to 
tional link by a bit in the parameter field. 45 receive the next frame group. 

The parameter field of the SRR request contains the In summary, according to an embodiment of the pres- 
largest of the skew values determined by the master ent invention, the skew rules are as follows: 
receivers. The master can also add a pad value in a All transceivers must start transmission of the link- 
similar manner as the slave (in sending the skew results control word (LC) within 2.5 word transmission times 
response). This value determines the link-control-word 50 across all of the drivers sending a frame group. A word 
to link-control-word spacing that is required for any transmission time is the amount of time it takes to 
frame groups that are to be sent by the slave. launch (transmit on to the link) a word of information 

Advantageously, the skew measurement method ac- (32 data bits plus 8 bits of encoding information in the 
cording to an embodiment of the present invention can present embodiment). 

fec*^™ t0tal skew including the systemic 55 Start-to-start spacing of link^ntrol words of back- 
(SSKEW) and random (RSKEW) elements thus reduc- to-back frames on a single driver may be no closer than 
ing or eliminating the need for pad values depending on the value that is established during the skew test proce- 
the confidence level required. In such an embodiment, dure. The start-to-start frame spacing is contained in the 
the operation of each carrier at a given skew value (in establish-transceiver-group request or the largest skew 
the skew register 594) is verified by repeating the skew 60 value returned in the skew results response frames 
test a multiplicity of times (e.g. 100) at that value. If a Alternatively, the start-to-start frame spacing is the 
earner can not pass the skew test even once (i.e. a repli- largest skew value for the operational link contained in 
cated frame is not received within the tested skew win- the skew results request or the largest skew value for 
dow), it is considered to not be within the tested skew the operational link returned in the skew results re- 
window as indicated by the skew register. 65 sponse frames. 

A summary of the skew test/measurement procedure If a link control word is received on a single receiver 
as viewed from a receiving end of the link is illustrated with less separation from the previous link control word 
in FIG. 7 and FIG. 10. Once the receiver unit is in skew than the start-to-start frame spacing as determined 


04/27/2004, EAST Version: 1.4.1 


5,425,020 

17 18 

above, the frame is discarded as described by the func- 568. Writing into the FIFO 568 causes an interrupt to 

tion of OR gate 592. the microprocessor. The microprocessor checks the 

Transceivers check to see that link-control words for summary information to see if all the frames of the 
each frame group are received within a predetermined frame group have been received. If all of the frames 
time span from arrival of the first frame on the link. The 5 have been received, the values in the SKEW 
length of the time span is specified by the maximum 501(A,B,C,D) registers is saved in the microprocessor's 
measured (worst case) skew value as determined by the local store before a validity test of skew data is made, 
skew test and can be optionally padded with a model FIG. 11 shows the Skew Table 1102 which is kept in 
dependant value that allows for long term jitter. The the microprocessor's local storage, and there is an entry 
checking is performed only on the transceivers that are 10 for each of the four receivers. For each receiver, there 
the members of the intended link as defined by the ETG is a minimum (MIN) 1104 and maximum (MAX) 1106 
request frame group sent or received and recognized. If variable, and each of these variables is an 8 bit number 
the alternative method is used, the checking is per- having possible values from 0 to 255. As skew measure- 
formed only on the transceivers that are the members of ment frames are received, the values in the Skew Table 
the intended link as defined by the skew results request 15 1102 are processed by the microprocessor, 
and skew results response. The skew measurement test proceeds in exactly the 

If a frame group is received with a skew error, the same way as shown in either FIG. 6 or the alternate 

frame group is rejected, or discarded as appropriate. method shown in FIG. 9. The same skew-measure- 

Transceivers that fail a skew check are logged out for request and skew-measure-response frames are used as 

maintenance purposes. 20 the calibration frames with the improved method; how- 

In accordance with the improvements described ever, many fewer frame exchanges are needed to 

herein, FIG. 5A illustrates the additional registers used achieve the same accuracy as are required by the previ- 

by the transceiver receiver in monitoring skew. The ous methods. The flowcharts shown in FIGS. 7, 8, and 

process of skew measurement utilizes a new method 10 are no longer used. 

employed in the transceiver receiver, as illustrated by 25 FIG. 12 is a flowchart of the improved skew mea- 

FIGS. 11-13. surement code based on the frame exchange shown in 

The improved method of measuring skew required FIG. 9. The frame exchange shown in FIG. 6 would 

development of new hardware, starting with adding result in a procedure that is largely the same as the 

SKEW registers (SKEW0 501A, SKEW1 501B, procedure described here. The same procedure is used 

SKEW2 501C, and SKEW 501D) to the hardware 30 by both ends of the link with the exception of the han- 

shown in FIG. 5. These SKEW registers are used to dling of the first skew measurement frames. These dif- 

measure the relative skew between frames of a frame ferences are described below. In the Initialization 1202 

group and are shown in FIG. 5A. Each of these regis- step at the beginning of the skew measurement proce- 

ters has a single input from the SCNTR 518 whose dure, all of the receivers are active, the SKEW 594 

operation is described earlier. The outputs of the 35 register is set to 255, and the Skew Table 1102 is initial- 

SKEW 501 (A,B,C,D) registers are connected to the ized. The MIN 1104 variables are set to 255 and the 

microprocessor. The gating in of the SKEW MAX 1106 variables are set to zero. Initializing the 

501(A,B,C,D) registers is controlled by lines MIN 1104 and MAX 1106 variables to these values may 

510(A,B,C, D), respectively. Each time a new LC is seem contradictory, but as demonstrated below, the 

received by RCV0 502, RCV1 504, RCV2 506, and 40 intended use of the MIN 1104 variable is to determine 

RCV3 508, lines 510(A,B,C,D), respectively, are active the minimum skew value as the skew procedure pro- 

for 1 cycle. As a result, the corresponding SKEW 501 ceeds and the intended use of the MAX 1105 variable is 

(A,B,C,D) register gates in the value in the SCNTR 518 to determine the maximum skew value as the skew 

register at the time the LC is received. procedure proceeds. The MIN 1104 variable starts at 

The operation of the SKEW 501(A,B,C,D) registers 45 the largest value and is reduced as the skew values are 

is as follows. Before the first frame of a frame group is processed, and the MAX 1106 variable starts at the 

received, the SCNTR 518 has already been reset to smallest value and is increased as the skew values are 

zero. When the first frame of the frame group is re- processed. 

ceived, the corresponding SKEW 501 (A,B,C,D) regis- The Send Measure Frame Group 1204 step is exe- 

ter receives the zero value from the SCNTR 518. As 50 cuted from the Initialization 1202 step only if this side of 

subsequent frames are received, their corresponding the link has the higher ND. If this side of the link has the 

SKEW 501 (A,B,C,D) registers receive the skew values lower ND, it waits for a skew-measure-request 906 

from the SCNTR 518. After all the frames of a frame frame group from the higher ND side of the link. In the 

group have been received, the SKEW 501 (A,B,QD) Frame Received 1206 step, the code is waiting fof an 

registers contain a measurement of the relative arrival 55 interrupt caused by the receipt of a frame group. After 

times of the individual frames of the frame group. The a frame group is received (either a skew-measure- 

SKEW 501(A,B,C,D) registers that contain a zero request 906 or skew-measure-response 908), the code 

value indicate which frames arrived first. checks to see if all of the frame group has been received 

The SKEW 594 register is not decremented by the in the Frame Complete 1208 step. If all of the frame 

microprocessor as the skew measurement procedure 60 group has not been received, the microcode pauses in 

progresses, but is kept at a value of 255 throughout the the Wait 1210 step for about 1 millisecond before it 

entire procedure. Each time a frame group is received, decides that the missing frames of the frame group will 

the LC and summary information are stored in the never arrive. This one millisecond wait time allows for 

FIFO 568. If some of the frames of the frame group are a possible fiber optical cable length mismatch of up to 

damaged by noise, they are not recognized as frames 65 200 kilometers, enough time to account for any misca- 

and the SCNTR 518 eventually reaches a value of 255 bling. If a late frame is received within the one millisec- 

(equaling the value in the SKEW 594 register). The LC ond as determined in decision block 1209, the corre- 

and summary information is then written into the FIFO spending receiver is outgated in step 1211 and the mea- 
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surement procedure continues. If a late frame is never MIN 1104 variables are added to twice the random 

received, the microcode assumes that the missing frame skew (RSKEW) value in step 1356 to determine the 

was damaged by link noise, and either a frame-measure- value for the maximum skew. This method is used to 

request or frame-measure-response frame-group is sent determine the maximum skew because even with multi- 

to the other side of the link depending on the weather 5 pie frame-group exchanges, there is a finite probability 

this side of the link has the higher or lower ND. By that the maximum skew value may not occur during the 

assuming that one or more of the frames was lost, the test. With this method, multiple frame-group exchanges 

receiver is not outgated by a lost frame and is allowed to are used to find the smallest minimum skew so that the 

continue to participate in the skew measurement proce- required frame to frame times are minimized- After the 

dure. If the frame group is complete, the SKEW 10 maximum values are computed from the MIN 1104 

501(A,B,C,D) registers are sampled in the Read Skew variables, they are checked for values greater than 127 

Registers 1212 step, and the sampled value is stored in word tinics m dec ision block 1358. If any are larger than 

li ni( L? pr0 ^ eSSOr,S locaI stora S e * 127, their corresponding transceivers cannot be used to 

The Skew Regs Still Valid 1214 step is a test to ensure fonn ^ operational link. In this case, the SKEW value 

that exactly one frame has been received from each 15 is re pi ace d by a value of 255 in step 1360. 

T Ef? f* 1 ?^!™"* iS I ? dve * value m Returning to FIG. 12, the Send Results Frame 1228 

the SKEW 501 (A.B.C D) registers is meanmgless since st sends the vaIues computed in ^ Process Table 

the additional frame changed to value in the SKEW st n26 [n the skew . results . reques t 922 and skew- 

• Cgl T v! ^ b S£^ t ^l^c^ I , results-response 924 frames. Recall that any values that 

ing the V0 528, VI 530, V2 532, and V3 534 bits for a 20 i ^ , n -r i j i nee 

, f j ' . ff . are larger than 127 are replaced with a value of 255 

value of zero and making sure that the FIFO 568 is • *f *u * *u 

™ * Tr *t. r*t. i • • ^ indicating that the corresponding receiver was either 

empty. If either of these conditions is not met, the other . „ A e , . i 

*u r t. v * r ii * i * outgated or had excessive skew, 

end of the link is not following the skew measurement *v \. c *v i M n r *u 

^ #A-Al : *i. * ■* * jjv * r i r- , After the exchange of the skew results frames, the 

protocol in that it sent an additional frame before al- . . r . , - . • , , ... \. 

L lia j ^ u„ * i fu * * • *u establishment of the link proceeds as desenbed earlier. 

lowed to by the protocol. The test is then restarted from 25 , , c , . ,. 

the besinnine While we have desenbed our preferred embodiments 

The Update Table 1218 step is shown in more detail ' ? f ° ur ^ticn. it will be understood that those skilled 

in FIG. 13A. For each receiver, the values obtained » both ™ w * the future may make various 

from the SKEW 501 (A,B,C,D) registers are compared ^provements and enhancements which fall within the 

to the MIN 1104 and MAX 1106 variables in the Skew 30 SC0 P e of ^ claims which follow * These claims should 

Table 1102. First, if the skew value is less than the MIN be «»*nicd to maintain the.proper protection for the 

1104 variable (shown in step 1302), the contents of the m \™* on ^ st . ^^sed. 

MIN 1104 variable is replaced with the skew value What 1S claim ea is: 

(shown in step 1304). Otherwise, the MIN 1104 variable 1 A method for measuring and verifying the skew 

is left unchanged. Second, if the skew value is more 35 ^tween frames of a frame group over a communica- 

than the MAX 1106 variable (shown in step 1306), the tions link havin S a plurality of carriers, comprising the 

contents of the MAX 1106 variable is replaced with the stc P s of: 

skew value (shown in step 1308). Otherwise, the MAX receiving, in parallel, a series of calibration frames 

1106 variable is left unchanged. The above procedure over of ^ carriers; 

repeats for the other configured receivers (as many as 40 measuring the relative arrival times of said calibration 

three) before the Update Skew Table 1218 step com- frames; 

plctes. constructing a skew table having measured skew for 

After the Update Skew Table 1218 step completes, a e3X ^ 1 carrier; 

counter indicating the number of successful skew-meas- updating said skew table as successive measurements 

ure-request or skew-measure-response frame-groups 45 made by receiving successive calibration 

that have been received is incremented in the Increment frames; 

Count 1220 step. This count is tested in the Done 1222 computing the maximum skew from said skew table; 

step, and control is passed to the Send Measure Frame ai *d 

Group 1204 step if more tests are required. The Com- communicating said computed maximum skew to the 

plete 1224 step checks if the other side of the link has 50 opposite end of the link. 

received enough skew measure frames. If it has not, 2. The method of claim 1 wherein said measuring 
control is passed to the Send Measure Frame Group comprises recording in a group of registers the relative 
1204 step. If the other side of the link has completed, the arrival times of the calibration frames as they are re- 
microcode the entire skew test has completed. ceived. 

The Process Table 1226 step is shown in more detail 55 3. The method of claim 2 wherein each of said regis- 

in FIG. 13B. First, all of the variables are converted ters is set to the value of a corresponding skew counting 

from cycle counts to link word times in the Convert register as said calibration frames are received. 

1350 step. Next, in decision block 1352, for each re- 4. The method of claim 1 wherein the skew table is 

ceiver the MIN 1104 variable is subtracted from the constructed at the beginning of the skew measurement 

MAX 1106 variable. If the difference is greater than 60 procedure and has one set of entries per receiver, 

twice the random skew value (RSKEW) allowed by the 5. The method of claim 4 wherein said skew table 

system, an error 1354 is signaled. In this case, the link is entries are initialized with a maximum value in the vari- 

considered unreliable, and operations cease. In actual able that records the minimum value for the measured 

practice, this difference is computed during testing in skew and a minimum value in the variable that records 

engineering test of the system in a laboratory environ- 65 the maximum value for the measured skew, 

ment when the computation is made many many (e.g. a 6, The method of claim 1 wherein said skew table is 

million) times. If the system is designed correctly, this updated as successive calibration frames are received 

difference is always within acceptable bounds, and the such that a comparison is made between the measured 
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value and the corresponding value in the skew table for 
each of said carriers further comprising the steps of: 

comparing the measured skew value to the corre- 
sponding skew table minimum value; 

replacing the skew table minimum value if said com- 5 
paring reveals that the measured skew value is less; 

comparing the measured skew value to the skew table 
maximum value; and 

replacing the skew table maximum value if said com- 
paring reveals that the measured skew value is 10 
more. 

7. The method of claim 1 wherein the final skew 
values are computed from the skew table at the end of 
the skew measurement procedure further comprising 
the steps of: 15 

adding twice the value of the random element contri- 
bution to the skew to the minimum numbers in the 
skew table; and 

replacing values over 127 with 255 to indicate that 
the receiver is outside of the allowable skew limit. 20 

8. The method of claim 1 wherein in a manufacturing 
process for computing a maximum random skew, all 
cycle count variables for measuring skew are converted 
from cycle counts to link word times, 

and, in a decision block for each receiver a MIN 25 
variable is subtracted from a MAX variable, and if 
the difference is greater than twice the random 
skew value (RSKEW) allowed by the system, an 
error is signaled, in which case, the link is consid- 
ered unreliable, and operations cease. 30 

9. A transceiver for sending and receiving message 
frames, comprising: 

means for coupling said transceiver to a communica- 
tions link; 

transmission section for asynchronously transmitting 35 
message data in frame groups across multiple car- 
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rier conductors of the link and for coupling the 
frame groups together; 

a receiver section for receiving frames and for deter- 
mining which frames are part of a group and for 
determining a relative skew on a conductor of a 
link by performing skew measurements on a series 
of calibration frames sent by a transmitter on an- 
other end of said link; 

a monitor means for performing said skew measure- 
ments, including a plurality of skew measurement 
registers for measuring the relative skew between 
frames of a frame group, each register having an 
input from a skew counter, and outputs coupled to 
a microprocessor controlling the monitor means, 
each register gating in the value in said skew 
counter at the time a link control word is received; 

said monitoring means including means for gathering, 
each time a link control word is received for a skew 
measure calibration frame, skew information from 
all of the carrier conductors of the link, means for 
measuring the relative arrival times of said calibra- 
tion frames; 

means for constructing a skew table; 

means for updating said skew table as successive 
measurements are made by receiving successive 
calibration frames; and 

means for computing the maximum skew from the 
said skew table and for communicating said com- 
puted maximum skew to the opposite end of the 
link. 

10. A transceiver according to claim 9 further com- 
prising: 

means for determining that one or more frames of the 
calibration frames have been lost and have not 
exceeded the maximum measurable skew limits. 
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